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ABSTRACT - - 

In evaluation of the Elementary Secondary Education 
Act Title S compensatory education prcgramsr both the federal 
reporting mandate (using formal evaluatioiil and local needs (using 
informal evaluatichj must _be served. In Chicago Public Schools^ 
computerized processing of standardized test scores allows the 
reporting requirement to be met with a minimum of staff and time- The 
bulk of staff ahdtime are used to gather information aboutprogram 
implementation aiid conduct at ^ the local schools. This information is 
returned to school principals andpfqgram administrators several _ 
times yearly. The local evaluation is beginning to focus on _two sets 
of persons as its subject: (1) the student-teacher relationship which 
sets the limits to pupils*^ learning; and (2V the schooi principal and 
the various administrators who have_the power to promote_6r vitiate 
programs. The reality_in which these persons act fluctuates* 
Nevertheless r that varying reality is the context in which decisions 
about Title I programs are made. (Rt) 
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FORMAL AND__INFpRtlAL EVALUATION 
TO SERVE TWO MASTERS WELL 

Tfie form and content of educational evaluation 
have recently been much criticized, from within and without 
the profession i Policy makers and administrators contend 
that educational evaluation is of poor quality, lacks 
credibility, fails to address relevant issues, and arrives 
at the wrong moment (Bissell 197 9; Florio, Behrmann and 
Goltz 1979; Wisler and Anderson 1979) . Some practitioners 
claim that the dominant mode of evaluation fails to provide 
appropriate descriptions, that emphasis on the "straight 
facts" distorts the meaning of events {Eisner 1979a; Schwab 
1970; Stake 1976.) Others argue the technical details of 
alternative representations of the "facts" (Burstein and 
Miller 1978; Linn and Slinde 1977i) 

In part these criticisms reflect the individual perspec- 
tives, the different audiences, and the varying information 
needs of the commentators. Nevertheless, a fundamental 
question underlies the concerns raised. This has to do with 
the adequacy of the dominant tradition of educational 
research, that based on the desire to use the methods of 
science to explicate social phenomena. Great technical strides 
have been made within this tradition and rigorous canons of 
ethics and procedure control and protect the use of these 
methodologies. Still , most educational evaluators— at 
least those I know — share Eisner's (1979b, p. 11) uneasiness: 
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such approaches r and the methods of inquiry 
that are regarded as legitimate within their 
borders, somehow fail to tell the whoie_story. 
As a result of the partial view that such methods 
provide, a biased, even distorted picture of the 
reality that we are attempting to understand and 
improve can occur. 

The RMC models for the evaluation of ESEA Title I compensa- 
tory education programs (Tallmadge and Wood 1976) are the 
specific examples responsible for the unease about the 
form, content, and meaning of educational evaluation which 
I wish to discuss. These models aim to provide federal 
policymakers with useful information regarding the effec- 
tiveness of a myriad of special educational activities 
implemented by local schools. Those who devised these models 
appear to have understood the complexity of the task assigned 
to them; nevertheless, they remain controversial (Gamel, 
Tallmadge, Wood and Binkiey 1975; Kearns 1978; Linn 1979; 
Wiley 1979.) 

Title i programs are by definition local. Now matter 
how technically adequate, the upward aggregation of data 
must result in a loss of information. ^ The critical 
question is: are the data which survive the aggregation 
process meaningful? Since these are compensatory education 
programs, an indicator of improvement based on standardized 
achievement test scores—the much discussed Normal Curve 
Equivalent gaihs--was developed. Yet, at the local level, 
test scores are not always the most meaningful indicators of 
project impact. Knowledge of any group of pupils" gain scores 
is an inadequate indicator of program effect, as the local 
evaluator and admihistrator are all too v/eii aware; 
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To quote Eisner again: 

,,, There is, 6f course, nothing wrong with knowing 
how well or how poorly a student perforins. Yet 
sqhools, insofar as they are educational inititu- 
tions should not be content with perf orman5e. 
Education as a process is concerned with the 
cultivation of intellectual power, and the 
ability to determine what a student knov/s is not 
necessarily useful or sufficient for making that 
process more effective. 

There is a second cause for concern. The emphasis of 
the Title I evaluation has been on the implementation of the 
national reporting system. This has caused local evalua- 
tions to focus their attention on the technical and bureau- 
cratic problems of obtaining and reporting data to state 
Education agencies. This emphasis may have resulted in too 
iittifc thought and too little manpower being available for 
the local aspects of the Title i evaluation. Yet, any evalua- 
tion of programs that are locally designed and implemented 
must emphasize local conditions and circumstances to be 
meaningful. A brief history of Title I evaluation in ehicag6 
may illustrate this. 

The Title I project in Chicago is now more than a dozen 
years old. It annually serves almost 70,000 pupils in about 
300 schools in a district composed 6f approximately 600 
elementary schools enrolling roughly half a million pupils. 
Over 40 distinct educational activities are provided with 
Title i monies. Schools select from among these activities, 
as their pupils' needs and their funds permit. The initial 
emphasis of the local evaluation was on the effectiveness of 
the individual activities, paralleling the national emphasis 
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on project evaiuatibhs. Three phases are evident in the 
history of Title I evaluation in Chicago, 

The initial evaluation effort made use of outside 
contractors to evaluate the individual activities. The 
function of the Board of Education's Title I evaluators^ of 
which there were usually about eighty was principally one 
of liaison between the external contractors and local school 
and central office staff. The external contractors carried 
out almost all instrument development ^ data collection (with 
the exception of achievement testing which was done on a city- 
wide basis by the Board) , and report writing. As a consequence ^ 
Board personnel were only indirectly involved with the specific 
Title i programs and the schools in which they operated. 

The evaluation reports produced in this manner were of 
greatly varying quality. C3iven numerous external contractors y 
uniformity of reports was extremely limited and comparisons 
among programs generally impossible; Publication was fre- 
quently delayed: in some cases reports were not available 
until several years after the close of a school year; Dupli- 
cation of data collection procedures was common^ Rectification 
of procedural errors was difficult i And hot lea^t^ the con- 
tracted evaiuatbrs were often far removed from the day-to-day 
developments in schools and programs/ forcing reports of such 
generality that their utility became questionable. The required 
reporting to the state educational agency , meanwhile ^ took the 
form of transmitting a computer tape of the test results for 
^ Title I pupils obtained as a part of the annual city-wide 
spring testing. Analyses of these data at the local level 
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was originally minimal but came with time to be the primary 
thrust of the evaldation. 

Iri part in response to the obvious inadequacy of these 
procedures^ the Department of Research and Evaluation was 
formed as a semi-independent branch of the central administra- 
tion , reporting directly to the general superintendent. As 
quickly as possible, the practice of using outside contrac- 
tors was discontinued. Eight evaluators were assigned to 
a Title I evaluation unit. Each evaiuator was responsible 
for the evaluation of five or six Title I activities. The 
Department also instituted a series of innovations designed 
to expedite data collection, improve comparability, provide 
more detailed and comprehensive evaluation reports, and ensure 
more prompt evaluations. Data collection forms were reduced 
in number. Two page interim reports—necessitated by the 
continued delay of the final reports- — -were provided to assist 
schools in the selection of activities. Considerable financial 
savings were also realized. For instance, in 1976-77 the 
externally contracted Title I classroom observation study 
cost roughly $40,000. The following year, using Board of 
Education staff, the cost of this aspect of the evaluation 
was reduced to under $15,000. The time for the evaluation 
effort decreased from one year to less than six months. 

The central purpose of the evaluation remained the 
identification of the most effective activities as determined 
by standardized test results. This task required the comparison 
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of activities enrolling pupils of varying age groups and 
ability levels, A method of transforming achievement test 
scores, similar to that propounded in the RMC models but 
developed independently, was chosen. Within age-cycle 
distributions of pretest and posttest scores were indivi- 
dually standardized and differences in the distributional 
placement of Individuals at pretest and posttest times were 
compared. These data confirmed that in terms of test 
performance most activities pfodctced approximately the same 
degree of pupil growth. Each year only two or three activi- 
ties could be identified to be more effective than others. 
There was little consistency from year to year cbhcerhihg 
relative effectiveness: a program appearing effective one 
year often demonstrated only average results the following 
year . 

From this a set of intertwined conclusions was drawn: 
the efficiency of Title i activities in Chicago depends more 
upon which schools select activities, which pupils schools 
choose to participate, how individual schools implement 
activities, and the amount of staff training the program 
vendors supply. Each of these conditions could and did change 
from year to year. Regardless of the precision with which 
an activity was specified, schools could be counted upon to 
modify the implementation of an activity along lines most 
appealing to the local school staff. Clearly, under such 
conditions the focus on pupils* academic achievement within 
activities could hot be expected to provide a helpful evalu- 
ation. 

8 
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In conseguencer the Department in 1978-79 revised the 
evaluation effort once more. Two principal improvements were 
sought: near immediate reporting and school level evaluation. 
Nine Title I evaluators were each assigned to a specific 
set of schools. Most of their time was spent in these 
schools observing and assisting local school Title I staff. 
The evaluators were required to prepare anecdotal summaries 
of each visit, to obtain certain minimal information on 
program enrollments and characteristics, and to conduct 
classroom observations and staff interviews. Given a suffi- 
cient level of professionalism and trusty it was expected 
that the field evaluators would be able to obtain evaluative 
information even as they provided feedback to the local 
schools in the attempt to improve the conduct of Title I 
instruction. At the Department of Research and Evaluation 
a staff of three or four persons monitored field events > 
coordinated and interpreted the information received^ wrote 
the final reports^ arid prepared brief reports on special 
topics for immediate dissemination to the schools and district 
staff. 

From the point of view of the Department, this approach 
greatly improved the Title I evaluation. Information on the 
local schools arid the Title I activities they salected has 
become much moire detailed and aririual and periodic reports 
seem much more informative and useful. School arid central 
office administrators appear to concur in this assessment 
(Hamilton 1980). Nevertheless, problems remain. Not the 
least of these is tension within the staff concerning the 
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proper role of evaluation. Some see a role based bh 
evaluation of "hard" data. They interpret the federal 
mandate and the RMC models as the primary responsibility. 
The majority contend this duty is a minimal requirement 
and that a more thorough interpretation and understanding 
must be reached. 

Although I have no wish to add another neologism^ 
let alone one that is dichotbmousr to the stock of jargon 
current in educational evaluation, i would like to make use 
of a pair of labels as intellectual shorthand. The nature 
of the uneasiness I have referred to can best be understood 
by distinguishing between formal and informal evaluation. 
This dichotomy is in many ways similar to others adumbrated 
in the recent past: Schermerhorn and Williams' (1979) 
preordinate and responsive approaches. La Belle, Moll and 
Weisner's (1979) juxtaposition of input-output and participant 
researcher strategies, Schwab's (1970) theoretic and practical 
orientations, Getzels and Suba*s (1957) distinction between 
the nomothetic and the idibgraphic, stnnmative versus formative 
evaluation (Scriven 1967) , tb name a few. I will make no effort 
to spell out precisely the phiibsbphical differences nor trace 
the history of these concepts in the sbcibibgy of knowledge 
of evaluation, although elements o£ the tension between the 
empirical and phenbmenolbgical schbbls of social analysis 
are evident. The empiricist attempts tb uncover sbciai 
"truths" and uses them to determine consequences; the 
phenomendlogist is more interested in comprehending "meanings". 
The former holds "laws" tb be the primary object of inquiry. 

10 
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the latter assigns priority to persons arid iriteritibris, 
I Wish to emphasize that I do not iritend to deriigrate ariy 
method of analysis or school of evaluation; what I hope 
to accomplish is a specification of usefulness arid proper 
place. 

Formal evaluation stresses scieritific methods arid 
numerical techrioiogy . It mairi tools are descriptive arid 
inferential statistics and the discipline of expeirimerital 
design and analysis. The intent of formal evaluation is 
the uncovery of reproducible educational outcomes, particu- 
larly student outcomes. It aspires to objectivity and 
neutrality. Formal evaluation reports, while they may be 
descriptive r prefer to elicit the laws which govern behavior. 
In this attempt sophisticated statistical methods are called 
upon.'' The complement these methods, equally sophisticated 
measurement of the traits of interest are sought. Since 
distributions of students* scores on standardized tests often 
have most of the properties these methods require, they ai e 
frequently the subject of formal evaluations Conclusions are 
generally stated with hesitancy and reticence and are much 
hemmed about with qualifications Formal evaluation is the 
dominant mode in educational research and evaluation todays 
domiinant both in being the professional ideal to which most 
evaluators strive and in being the form most evaluation 
reports take. 

Although many evaluators now call for evaluatiohis 
(ie^ighed to improve educational processes, the history of 

11 

o 

ERIC 



ID 

evaluation and the training of most evaluators have empha- 
sized summation, classification, and technique. Despite 
current rhetoric, the 1971 statement of Bloom, Hastings, 
and Madaus remains true: "As testing and other forms of 
evaluation are commonly used in the schools, they contri- 
bute little to the improvement of teaching and learning" 
(P- 7). The evaluator's own desires for advancement urge 
him to use formal methods: the professional journals appear 
more likely to publish papers displaying traditional research 
designs and sophisticated numerical skills. Paradoxically^ 
it is often easier to author a statistically sophisticated 
report than to write a clear and meaningful evaluation using 
logic, vocabulary, and the rules of grammar as tools* 

The RMG models for the evaluation of Title I programs 
fall squarely within this tradition of formal evaluation. 
Given the concerns of administrators and legislators at the 
federal level, this may be appropriate i Summative results 
appear to be useful for yes-or-ho decisions regarding funding 
or its continuation • 

At the local level emphasis is on improving instruction, 
usually by modifying programs already in place. Local resources, 
for a variety of reasons^ simply do not permit continual 
picking and choosing amiong programs and curricula. Informa- 
lity, improvement, feedback, formative evaluation-- these are 
the lif eblood . of local evaluation. The RMC models do little 
to improve circulation; they are designed to indicate only 
whether no or major surgery is required. More generally. 
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the formal ernpiricist approach has limited utility Ibcaiiy. 
Often properly trained staff aire unavailable. Outside 
contracts are expensive and as iri Chicago^ may not be able 
to capture the flavor of local events. A more informal, 
phenomenoiogicai approach may be more suitable to local 
needs. Events have a way of marching onward, oblivious to 
the needs of the researcher. Formal evaluation has little 
capacity to adapt to the succession of events. informal 
approaches, however, can capitalize on this. 

1 would like to illustrate these concerns using the 
progression of evaluation schema just described for Chicago. 
The person most capable of exerting control over the implemen- 
tation and modification of instruction is the local principal. 
Second in influence is probably the Title I coordinator > whose 
task it is to assist Title I teachers. Much further down the 
list is the local evaluator. While he often feels he has 
information of great usefulness to principals and coordinators, 
he finds himself ignored and ineffectual. Why? 

The Title I evaluator operates on a basis much different 
from that which guides the principal. In many ways the prin- 
cipal is isolated from the circumstances determining the be- 
havior and orientation of central office staff. He understands 
his school, his staff, and his students. Principals tend to 
share the conviction that they know their own territory best. 
Orders, requests ^ even information from other sources in the 
educational bureaucracy all are, to varying extents, unwelcome 
infringements on the business of running a school. Evaluation 
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is perceived to be legitimated by outside sources. Principal 
quickly perceive that much of the testing and form-filling is 
not intended to help them, but to fulfill other obligations. 
Catering to the evaluator's needs is equivalent to obeying 
outside directives. This reduces the principal's perception 
of his own authority. Especially as regards Title i, where 
the major justification for evaluation is a federal or state 
requirement, this_Broduces only grudging cooperation. Reports 
of personal and professional recalcitrance were not infre- 
quently made by the external contractors in the first phase 
of Chicago's Title I evaluation. With the switch to school- 
focused evaluation such complaints still arise , though more 
often based on personality than on professional differences. 

These tendencies are exacerbated by emphasis on formal 
modes of evaluations Standardized testing of pupils and compl 
tion of rigid questionnaires by staff limit the principal's 
autohomy and authority; He cannot control the responses his 
pupils or staff makei Even if he wished wholeheartedly to 
improve pupil performance oh tests , he knows that current 
educational technology is insufficient to cause dramatic 
change i His pupils' test performance , particularly from 
year to year, is essentially fixed by the quality of the 
students it is his lot to lead. He knows he can influence 
other characteristics of his students and his school^ but thes 
are hot usually measured or evaluated^ certainly not wherever 
the BMC models set the tone for the evaluation. Despite his 
best efforts, the evaluation > as he sees it, will make cdnclu- 
sibhs and recbitimeildatibns which he can do little to implement. 
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Chicago's Title I evaluation reports at first 
emphasized the interpretation of mean scores of indivi- 
dual Title I activities but said little about the perfor- 
mance of pupils at individual schools. Given that a 
principal was convinced that a particular activity worked 
well at his school, he was likely to shrug off a report 
that the activity mean was below average. After all, some 
schools must be below and others above the average. So 
what if most schools reported difficulty in implementing 
Computer-Assisted instruction? Once the computers were 
properly connected, the CAI classroom at his school became 
an active, happy learning place. 

To the extent analysis of data is done along formal 
lines, the principal may not even be able to comprehend the 
methods the evaluator uses to come to his conclusions. Such 
technical intimidation cannot cause him to react favorably w 
Most principals are trained by their experience to mistrust 
what they cannot understand. Talk of NCEs arid multivariate 
regression is foreign to him i it does not appear to relate 
to more urgent problems: How can I improve teacher X's 
instruction? What attitude do I take to the parents of 
Johnny to get him to school more regularly? In Chicago > 
staff 6£ the Department of Research and Evaluation might 
well appreciate the rigor and elegance of some consultant' s 
analysis, but in the schools the analysis was often incomprehen- 
sible and therefore threatening and untrustworthy. 

ERIC 
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Clearly this litany coUld be extended. Nor is it 
unfamiliar. The standard panacea^ often espoused by the 
Technical Assistance Centers created by the ESEA legisla- 
tion, is to ihservice^ involve^ and educate local staff. 
Given time^ this may wbrk--especially if the TAGS shift 
their focus from the local evaluatdr to the local principal. 
In the meantime^ informal approaches to evaluation may create 
a better environment and more meaningful evaluations. 

By advocating more informal evaluations, I do not mean 
to suggest that proven techniques be abandoned or that the 
rigor with which evaluations are conducted be diluted. Rather, 
greater involvement in evaluation, particularly in evaluation 
design and interpretation, should be sought from those persons 
who have a real stake in the success of Title I programs: 
the local principal, coordinator, and program vendor. The 
RMC models should not intrigue the local evaluatdr so much 
that he retains too little enthusiasm for providing informa- 
tion useful to local principals and others with the capability 
to improve the content of instruction in Title I. 

informal evaluation requires the evaluator to take a 
more creative, more critical stance than does formal evalua- 
tion i Neither, to be effective, permits sloppy workmanship. 
The call for more informal evaluation is not a retreat from 
rigor nor a simplification of what evaluators do. It requires 
increased attention^ better observation^ and more accurate and 
useful description. 

In the evaluation of Title I programs^ two masters must 
be served: the federal reporting mandate and local needs. 
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In Chicago this is becoming possible. Computerized 
processing of standardized test scores allows the reporting 
requirement to be met with a minimum of staff and time. The 
bulk of staff and time are used to gather information about 
program impiementation and conduct at the local schools. This 
information is returned to school principals and program 
administrators several times yearly. In fiscal 1981 this 
feedback is to become almost instantaneous and is to occur 
continually. Program vendors are expected to become more 
active participants in the infotination flow. The Title I 
evaluation design facilita*^es this flow but also allows the 
retention of information for more traditional and formal end- 
of -year evaluation reports for schools, projects, and the 
overall program. 

The Title I evaluation staff is working ihcfeasihgly 
closely with principals and others — both at scheduled meetings 
and in informal consultations — to learn what information is 
desired and useful at the schools. Rules cf procedure and 
conduct are being established jointly beforehand (although 
much progress remains to be made) and informal presentations 
of findings are being made at the close of each school visit 
and at frequent intervals to staff not located at the schools. 
It is expected that some principals will continue to resist 
these efforts, if only because their time is limited. Never the 
less, the emphasis on personal contact and direct involvement 
is beginning to produce improved instruction. 

This emphasis on personal contact arid involvement is the 
central tenet of the emerging rationale for the form of Title I 
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evaidatibn in ehic^ago. The evaluation is, in the informal, 
phencrnj3holog teeil ^ense, beginning to focus on two sets of 
perK02:s. The first is composed of the student-teacher dyad 
as subject of the evaluation. It is their relationship which 
sets the limits to pupils' learning. The second set is 
composed of the school principal and the various adminis- 
trators as the object of the evaluation. They have the 
power to promote or vitiate programs. 

The more typical formal, empiricist approach slights 
both subject arid object. Analysis of test scores and question- 
naire responses replaces the student- teacher relationship. The 
search for "laws" arid regularities of behavior overwhelms 
the interest of principals and adminis^trators. It also 
ignores the changes over time in these interests and relation- 
ships. The reality in which these persons act fluctuates, 
sometimes dramatically, often in short spans of time. Neverthe 
less, that varying reality, much of which is political in 
essence, is the context in which decisions about Title I 
programs are made. Remaihihg aloof of this context cari drily 
bode ill for the success — in the sense of impact and relevance- 
of any evaluation-. 
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NOTES 



1, That the models ask rib questions about the shape, content/ 
or implementation of these programs is disquieting. Without 
knowledge of what a program looked like "on the ground,'' any 
evaluation must be of dubious value. The assignment of a result 
to a non-program is a real risk. 

2* Although Chicago was not aware of the format of Title I 
evaluation in Cincinnati (Ahn, Barta and Rockwbod 1976) there 
appear to be several similarities, particularly concerning the 
local school's perception of the role of the Title I evaluator. 
Title i evaluators need to develop greater levels of trust and 
rapport with local school staff (Felix 1978 j Hamilton 1980) and 
school personnel must come to see the relevance of evaluation to 
their work (David 1978) . 

3w The reader sometimes gains the impression that analytic 
techniques are chosen more for their impressiveness than for 
their utility. The propensity in evaluation reports toward 
discussion of bias and better measurement is a consequence of this 
approach and sometimes seems to take precedence over the actual 
analysis . 
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